This commit makes the `make-memzip.py` utility more resilient in certain
usage cases, which weren't accounted for in the original source.

In some Linux distributions the `zip` command may not be available,
although `unzip` could be.  Unless the `zip` package is installed this
script will fail with a generic "file not found" error, which has been
addressed in these changes set.

Also, the `zip` command by default adds a `.zip` suffix to archives,
which wasn't accounted for.  So, running `./make-memzip -z z1 -c z2 .`
would fail, as the script would build `z1.zip` whilst the header
generator attempted to open `z1` instead.  This has now been fixed.

Signed-off-by: Alessandro Gatti <[email protected]>

This commit modifies the native modules build system to use different
directories for different architectures.

Previously, the natmod build system would unconditionally put all build
data into a single directory (usually "$(PWD)/build" unless overridden),
and the static library runtime cache into another directory
("$(PWD)/.mpy_ld_cache").  That works if building is always done for a
single architecture or if a full clean is performed before switching
architectures, which sometimes is not wanted (collecting the runtime
cache may take quite some time depending on the module complexity, for
example).

With these changes, both the build directory and the runtime cache
directories are marked with the name of the architecture they target, so
for example "$(PWD)/build" becomes "$(PWD)/build_x64" (the same happens
to the cache directory name).

Signed-off-by: Alessandro Gatti <[email protected]>

This commit makes a minor change to the native module build scripts, to
make the intermediate MPY file keep the same file name it will have once
the final pre-processing step takes place.

A MPY file will retain its file name as the first QSTR entry in the
file, and before these changes the name being written would be the one
used for the intermediate MPY output file (ie.
"$BUILDDIR/$MODNAME.native.mpy").  The `mpy-tool.py` preprocessing pass
would keep that string intact but rename the file, so things wouldn't
match in error tracebacks.

Signed-off-by: Alessandro Gatti <[email protected]>

This commit adds an option to the MPY linker to brand generated file
with a custom internal file name rather than the raw output path
provided in the command line.

MPY files, as part of their preambles, will contain a path name as the
first entry in the constant string pool.  However that may contain
unneeded data (ie. sometimes the plain file name is enough) or it may
have sensitive data in there (if an alternate build path is provided it
may contain product code names or client names which should probably not
be disclosed).  Until now there was no provision to let users use a
different internal path name.

A new option was added to `mpy_ld.py`: "--source-name" which, if passed,
will write the given string in the module file.  If said option is not
provided, generated MPY files will still carry the raw command line
output path but stripped of all leading path segments.

The example natmod makefiles have been updated to always provide that
option whenever the final file name on the filesystem wouldn't match the
internal MPY file name.

Signed-off-by: Alessandro Gatti <[email protected]>

MICROPY_PY_LWIP_PPP is used in only one place and is generally set to
the value of MICROPY_PY_NETWORK_PPP_LWIP. Remove the former.

extmod/lwip-include/lwipopts_common.h: Replace MICROPY_PY_LWIP_PPP
                                  with MICROPY_PY_NETWORK_PPP_LWIP.
ports/mimxrt/mpconfigport.h: Remove redundant MICROPY_PY_LWIP_PPP.
ports/rp2/mpconfigport.h: Remove redundant MICROPY_PY_LWIP_PPP,
                          default MICROPY_PY_NETWORK_PPP_LWIP to
                          MICROPY_PY_LWIP.
ports/stm32/mpconfigport.h: Remove redundant MICROPY_PY_LWIP_PPP.

Signed-off-by: Phil Howard <[email protected]>

mpconfigport.h: Default MICROPY_PY_NETWORK_PPP_LWIP to 1.

Signed-off-by: Phil Howard <[email protected]>

This updates the PATHS to include:
- everything under `extmod/`
- everything under `py/` (in case one day there are subdirs there...)
- everything under `shared/`

This makes sure that any C source files added anywhere under these
directories will be formatted.

Signed-off-by: Damien George <[email protected]>

These need to be formatted due to the parent commit.

Signed-off-by: Damien George <[email protected]>

This reverts commit 6552836.

Signed-off-by: Angus Gratton <[email protected]>

This seems like it's only really a problem on Debug builds, but
I think can't hurt to increase it on all windows builds.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <[email protected]>

This removes the last usage of mp_binary_set_val_array_from_int(). It
will be a little slower, but shouldn't be measurably so compared to the
ADC sampling.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <[email protected]>

If >1 unittest-enabled module is included, the results of the merged
module won't match (as it runs some previously registered tests again).

Signed-off-by: Angus Gratton <[email protected]>

This PR was originally a cherry-pick of CircuitPython commits 095c844,
d103ac1, c592bd6 and 8664a65.

However, substantial additional changes so the implementation has diverged
a lot from CircuitPython's:

- Keep CPython >= 3.11 defaults for int.to_bytes(), consistent
  with 80c5e76 and 0b432b3.

- Refactors to reduce the code size impact of this change. Various code
  paths are now funneled into new function mp_obj_int_to_bytes() and
  existing function mp_binary_set_int(), except where a simple assignment
  is used for performance reasons (i.e. array, moductypes).

- Keep the MicroPython behaviour of not overflow checking assignments to
  arrays, bytearrays, etc. - but enable overflow checks as part of
  MicroPython V2.0.

- Update tests to work with the new behaviour (similar to CPython) and add
  coverage for new code and some corner cases. Some tests are converted to
  unittest so we can easily verify both current and V2.0 overflow.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <[email protected]>

Generated `qstr.i.last` files (output from the "pp" phase) can be very
large due to inclusion of large HAL headers.  Yet only a small amount of
this data is actually needed by the "split" phase of `makeqstrdefs.py`.

This commit improves the situation by filtering out unneeded lines as early
as possible: in the "pp" phase the C preprocessor runs, its output is
parsed via a pipe connection, and only those lines that are needed are
written out to the `qstr.i.last` file.  When the "split" phase runs it then
sees only the necessary lines (it does further regex filtering to get
exactly the lines it needs for the given mode).

Windows already does this filtering (it does not use `makeqstrdefs.py` for
the preprocessor phase) in its `ConcatPreProcFiles` function, and that
was added long ago in 29c8c8a.

This change significantly reduces the output build size and improves the
build speed:

- stm32 BOARD=PYBV10: build output goes from 152M down to 32M, and build is
  10% faster (29s down to 26s).  Building all stm32 boards, the total build
  size goes from about 11GB down to about 2.5GB.

- rp2 BOARD=RPI_PICO: build output goes from 87M down to 39M, and build is
  about 5% faster (23s down to 21.6s).

- esp32 BOARD=ESP32_GENERIC: build output goes from 307M down to 230M, and
  build is about 6% faster (101s down to 95s).

Tested all three ports above, that the files in build/genhdr are equivalent
(except of course `qstr.i.last`), and that PYBV10 is binary equivalent
before and after this change.

Signed-off-by: Damien George <[email protected]>

This adds a new `--trace-output` option to `run-tests.py` which mimics the
same option already in `run-multitests.py`.  Using it, the output from the
test running on a board is printed to stdout as it is received.

This allows easier debugging of tests, to see the output in real time.  For
tests that have complicated run parameters, eg using via-mpy or needing a
target wiring script, running them standalone (eg with `mpremote`) is not
possible, so using `run-tests.py` becomes mandatory, hence the need to
easily see the output without having to view the result file separately at
the end.  It's also useful to watch tests that pass but are long running.

Signed-off-by: Damien George <[email protected]>

Similar to other classes in the `machine` module, `SDCard` is now there
for a port to use if it enables MICROPY_PY_MACHINE_SDCARD.

Signed-off-by: Damien George <[email protected]>

This is now provided by the common one in `extmod/modmachine.c`.

Signed-off-by: Damien George <[email protected]>

To unify this with other ports, and to enable the common `machine.SDCard`
entry in `extmod/modmachine.c`.

Signed-off-by: Damien George <[email protected]>

To unify this with other ports, and to enable the common `machine.SDCard`
entry in `extmod/modmachine.c`.

Signed-off-by: Damien George <[email protected]>

This aligns the stm32 port with other ports that have `machine.SDCard`.
The same `SDCard` class is used for this and the existing `pyb.SDCard`, and
its API matches the standard one.

Signed-off-by: Damien George <[email protected]>

Add a minimal `network.WLAN` CSI API for ESP32 builds with CSI
enabled. Also enable ESP-IDF CSI support in the standard generic
`ESP32`, `ESP32-C3`, `ESP32-S3`, `ESP32-C5`, and `ESP32-C6`
board definitions.

Signed-off-by: Francesco Pace <[email protected]>

Add a board definition for the Seeed Studio XIAO ESP32C3
(ESP32-C3 with 4MB flash).
The REPL is on the native USB Serial/JTAG interface,
leaving UART0 free for the application.

Signed-off-by: Pavel Revak <[email protected]>

Add a board definition for the Seeed Studio XIAO ESP32C5
(ESP32-C5 with 8MB flash and 8MB quad SPIRAM).
The REPL is on the native USB Serial/JTAG interface,
leaving UART0 free for the application.

Signed-off-by: Pavel Revak <[email protected]>

Add a board definition for the Seeed Studio XIAO ESP32S3
(ESP32-S3 with 8MB flash and 8MB octal SPIRAM).
The REPL is on the native USB interface, leaving UART0 free
for the application.

The same firmware also runs on the XIAO ESP32S3 Plus: the flash size is
auto-detected, and the Plus's extra pins (D11-D19) are included in
pins.csv, which does not affect the standard board.

Signed-off-by: Pavel Revak <[email protected]>

By default the new I2C driver is enabled for esp-idf version >= 5.5.2.

With v5.5.2 and up the the additional bus probing is not needed any
more. It was needed because transfers to a non-existing address caused
an error message to be printed instead of just returning an error code.
With esp-idf v5.5.2 it returns just an error code.
For 0-length writes as being used by i2c.scan(), an additional code path
using i2c_master_execute_defined_operations() is needed to avoid an
error message.

Tested with a Generic ESP32 and esp-idf v5.5.1, v5.5.2 and v5.5.4.
Tested as well using the new driver with v5.5.1.
Tested V5.5.2 with Generic ESP32, ESP32S2, ESP32S3, ESP32C2, ESP32C3,
ESP32C5, ESP32C6, ESP32P4. Driving an external device as I2C controller.

There is a slight inconsistency: reading 0 bytes from a non-existing
address return an empty bytes object and does not raise an error.
Writing 0 bytes to a non-existing address returns an error. That case
is used by i2c.scan().

Signed-off-by: robert-hh <[email protected]>

If the new driver is enabled for versions previous to v5.5.2, an error
is raised and compilation aborted.

Signed-off-by: robert-hh <[email protected]>

The `MP_SEEK_xxx` constants are defined in `py/stream.h` exactly for use
by `py/stream.c` so that it doesn't depend on the C library.

Signed-off-by: Damien George <[email protected]>

Instead of magic numbers + a comment.

Signed-off-by: Damien George <[email protected]>

Adds a Modern Windows console implementation that can write
raw UTF-8 bytes directly (like POSIX)

Improve the Legacy consol to use an incremental decoder to handle
split UTF-8 sequences within writes.

fixes #15228

Signed-off-by: Jos Verlinde <[email protected]>

RemoteCommand.wr_bytes(): Now correctly encodes strings to UTF-8 bytes
before calculating length.

Based on the suggestion by @felixdoer

Fixes issue #13055.

Signed-off-by: Jos Verlinde <[email protected]>

Signed-off-by: Jos Verlinde <[email protected]>

Reduces code duplication while allowing different configurations.

Signed-off-by: Jos Verlinde <[email protected]>

Add regressions tests for utf-8 and unicode.

Signed-off-by: Jos Verlinde <[email protected]>

When soft_timer_heap is NULL the expression &soft_timer_heap->pairheap
is undefined behaviour in both C and C++. This is harmless today only
because pairheap is at offset 0, so the expression folds to NULL, but
a struct reorder would corrupt the heap. The same pattern occurs in
soft_timer_insert(), remove(), handler(), deinit() and gc_mark_all().

This patch fixes the UB by introducing a safe pairheap accessor that
avoids the undefined behaviour of &e->pairheap when e is NULL, while
guaranteeing that pairheap is always the first member, via static assert.

Signed-off-by: iabdalkader <[email protected]>

Make the function machine_bootloader_rtc() available to other modules by
declaring it in mpconfigport.h, when selected via its #define.

This is needed for the Arduino Nano ESP32 board to be able to call this
function from the board_init.c file.

Signed-off-by: Luca Burelli <[email protected]>

The custom machine.bootloader() function in the Nano ESP32 board
currently calls the Arduino bootloader to allow DFU updates on the
board. However there are situations where the hardware bootloader may
still be useful, for example when a full flash is required.

Make it so calling `machine.bootloader(1)` will fallback to the default
bootloader; anything else will result in the one by Arduino being started.

Signed-off-by: Luca Burelli <[email protected]>

So it can be skipped if the target has unicode enabled but doesn't support
f-strings.  For example ADAFRUIT_ITSYBITSY_M0_EXPRESS.

Signed-off-by: Damien George <[email protected]>

Tests are run from within their directory, so extmod is actually two levels
up, not one.  Use `base_path()` to get the correct value, following how
it's done in `run-tests.py`.

Signed-off-by: Damien George <[email protected]>

Instead of "latin" which is unsupported, and now an error since commit
770bb3b

Signed-off-by: Damien George <[email protected]>

Newer versions of `esptool` (eg v5.2.0) require the `--chip` option to be
specified.  And older versions (eg v4.5) accept it, so this change is
backwards compatible.

Signed-off-by: Damien George <[email protected]>

Add a new c_module() function to the manifest API that lets a manifest
declare which user C module directories should be included in the build.
This pairs with the build system changes that pick up the resulting paths
and feed them into USER_C_MODULES.

The function validates the path exists, is a directory, and contains a
micropython.mk or micropython.cmake, and supports the usual \$(MPY_DIR),
\$(BOARD_DIR), \$(PORT_DIR), \$(MPY_LIB_DIR) substitutions. Unresolved
\$(VAR) tokens raise a clear error rather than failing later.

makemanifest.py gains --list-c-modules which prints one path per line so
the build system can pick them up (newline output handles paths that
contain whitespace on the cmake side).

Signed-off-by: Andrew Leech <[email protected]>

When MICROPY_FROZEN_MANIFEST is set, the build system runs makemanifest.py
with --list-c-modules to extract C module paths from the manifest and
appends them to USER_C_MODULES so they go through the same code path as
user modules specified on the command line.

The Make side lives in py/manifest.mk (included from py/py.mk) and the
CMake side lives in py/manifest.cmake (included from py/usermod.cmake).
Both share defaulting for the MICROPY_MANIFEST_* substitution variables
so a manifest can reference $(MPY_DIR) etc. consistently on either side.
The make-side extraction is gated on the build target needing it and on
micropython-lib being initialised, and surfaces makemanifest.py errors
via .SHELLSTATUS. The cmake side uses FATAL_ERROR to match.

Signed-off-by: Andrew Leech <[email protected]>

Move the MICROPY_USER_FROZEN_MANIFEST resolution to run right after the
board config (rather than near the end of CMakeLists.txt) so it's applied
before py/usermod.cmake is included; py/manifest.cmake (invoked from
usermod.cmake) needs the final MICROPY_FROZEN_MANIFEST value to extract
c_module() paths. Also quote FROZEN_MANIFEST when forwarding it from the
Makefile to cmake so paths containing whitespace round-trip correctly.

Signed-off-by: Andrew Leech <[email protected]>

Quote FROZEN_MANIFEST when passing it through IDFPY_FLAGS so paths
containing whitespace round-trip correctly to the cmake configure step;
py/manifest.cmake (auto-included via py/usermod.cmake) needs the
resolved value to extract c_module() paths from the manifest.

Signed-off-by: Andrew Leech <[email protected]>

The micropython.cmake referenced examplemodule.c but the source file is
named modexamplepackage.c. The subpackage example wasn't built via cmake
before, so the typo went unnoticed.

Signed-off-by: Andrew Leech <[email protected]>

Document the new c_module() function in the manifest reference, including
the FROZEN_MANIFEST prerequisite, additive interaction with USER_C_MODULES,
and the make-side whitespace limitation. Add a corresponding section to
the cmodules developer guide so the feature is discoverable from both
entry points.

Signed-off-by: Andrew Leech <[email protected]>

Provides tests/tools/manifest_c_module.py, a shared frozen manifest that
pulls in the example user C modules via c_module() and includes the
port's default board manifest via $(PORT_DIR). Wires it up in ci.sh so
rp2 (RPI_PICO2), stm32 (PYBV11) and esp32 (SPIRAM variant) each exercise
the c_module() manifest path alongside the legacy USER_C_MODULES=
command-line variable.

Signed-off-by: Andrew Leech <[email protected]>

The unix coverage variant set USER_C_MODULES from mpconfigvariant.mk to
build the example user C modules into the test binary. Move that setting
into the coverage manifest as a c_module() call to exercise the new
manifest API end-to-end as part of the standard coverage test build.

Signed-off-by: Andrew Leech <[email protected]>

`NVIC_SetPriority` is an inline function with separate paths for IRQ number
negative vs non-negative.  When called here to set the GPIO IRQ priority,
gcc sees that the IRQ number could potentially be NotAvail_IRQn=-128
(although in practice that never occurs).  Gcc then issues an
array-out-of-bounds error due to the negative code path only allowing down
to -16 for the IRQ number.

Fix that by forcing the IRQ number to be positive, similar to how stm32
does it.  That also allows the compiler to optimise away the non-negative
code path.

Signed-off-by: Damien George <[email protected]>

These variables are never used by the function.

Signed-off-by: Damien George <[email protected]>

MP_STREAM_POLL was previously just returning MP_STREAM_ERROR.

Signed-off-by: Damien George <[email protected]>

Signed-off-by: Damien George <[email protected]>

The F4 and F7 HAL have a volatile debug variable that is only ever set.

Signed-off-by: Damien George <[email protected]>

This reverts commit 4edef87.

The `mod_network_init()` call was already moved before `boot.py` in commit
7d77d3e.

Signed-off-by: Damien George <[email protected]>

Specifically, either a debug build or a non-stripped release build.

Default build config is stripped, so it's redundant to generate debug info
and then strip it from the final binary.

This saves about 10% compile time compared to passing -g, and more than 10%
compared to passing -ggdb3.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <[email protected]>

Requires a tweak in makeqstrdefs to not enable debug output from the
preprocessor, as -g3 adds a lot of extra macro definitions - some of which
are picked up by makeqstrdefs otherwise.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <[email protected]>

This variant has the smallest application partition in flash.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <[email protected]>